Skip to content

Add "run CI checks" trigger phrases to generate-sdk-locally skill#15495

Open
Copilot wants to merge 3 commits intomainfrom
copilot/fix-ci-checks-azure-health-deidentification
Open

Add "run CI checks" trigger phrases to generate-sdk-locally skill#15495
Copilot wants to merge 3 commits intomainfrom
copilot/fix-ci-checks-azure-health-deidentification

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 5, 2026 •
edited
Loading

When users asked to "run CI checks" or "validate a package passes all checks", the LLM bypassed MCP tools entirely and ran raw dotnet commands directly. The skill's WHEN: triggers had no CI-check vocabulary, so it was never activated.

Changes

  • Skill description (WHEN: clause) — Added CI-oriented triggers: "run CI checks", "validate CI checks", "passes all checks", "run all checks", "validate package" to both the distributed skill (.github/skills/azsdk-common-generate-sdk-locally/SKILL.md) and its canonical source (plugins/azure-sdk-tools/skills/generate-sdk-locally/SKILL.md)

  • Steps section — Added a callout at the top clarifying that "CI checks" = build (azsdk_package_build_code) + validate (azsdk_package_run_check) + test (azsdk_package_run_tests), and that CI-only requests should skip steps 1–5 (generation)

  • Examples — Added the verbatim prompts from the bug report as canonical examples:

    "Can you run the CI checks for Azure.Health.Deidentification"
"Can you validate Azure.Search.Documents passes all checks?"
"Run all checks for the Azure.Storage.Blobs package"

  • Disambiguation — Changed DO NOT USE FOR: CI pipeline configuration → CI pipeline YAML configuration to prevent the LLM from misclassifying "run CI checks" as excluded

  • README trigger tables — Updated both README.md files to surface the new CI-related triggers

Copilot AI linked an issue May 5, 2026 that may be closed by this pull request
[Bug Bash] potential gap - if you ask the LLM to "run CI checks" it doesn't use the MCP at all #15442
Open
Copilot AI changed the title [WIP] Fix CI checks for Azure.Health.Deidentification Add "run CI checks" trigger phrases to generate-sdk-locally skill May 5, 2026
Copilot AI requested a review from l0lawrence May 5, 2026 18:08
@l0lawrence l0lawrence marked this pull request as ready for review May 5, 2026 19:39
Copilot AI review requested due to automatic review settings May 5, 2026 19:39
Copilot AI reviewed May 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the generate-sdk-locally shared skill so that CI-oriented user prompts (e.g., “run CI checks”, “passes all checks”) correctly trigger the skill and route the agent toward using the Azure SDK MCP tools instead of running raw local commands.

Changes:

  • Added CI-check vocabulary to the skill WHEN: triggers (and surfaced some of them in the skills READMEs).
  • Added “CI checks only” guidance plus new example prompts reflecting the reported issue.
  • Clarified the exclusion text to “CI pipeline YAML configuration” to reduce misclassification.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File Description
plugins/azure-sdk-tools/skills/README.md Updates the trigger table to include CI-related phrases for the generate-sdk-locally skill.
plugins/azure-sdk-tools/skills/generate-sdk-locally/SKILL.md Expands WHEN: triggers, adds CI-only guidance, and adds CI-check examples.
.github/skills/README.md Mirrors the trigger table update for the distributed .github/skills copy.
.github/skills/azsdk-common-generate-sdk-locally/SKILL.md Mirrors the skill trigger + CI-only guidance + examples for distribution.

Comment thread plugins/azure-sdk-tools/skills/generate-sdk-locally/SKILL.md

## Steps

> **CI checks only** (build + validate + test without regeneration): If the user asks to "run CI checks", "validate" or "check if a package passes all checks", skip to step 6. CI checks = build (`azsdk_package_build_code`) + validate (`azsdk_package_run_check`) + test (`azsdk_package_run_tests`).
Comment thread plugins/azure-sdk-tools/skills/generate-sdk-locally/SKILL.md

## Steps

> **CI checks only** (build + validate + test without regeneration): If the user asks to "run CI checks", "validate" or "check if a package passes all checks", skip to step 6. CI checks = build (`azsdk_package_build_code`) + validate (`azsdk_package_run_check`) + test (`azsdk_package_run_tests`).
Comment thread .github/skills/azsdk-common-generate-sdk-locally/SKILL.md

## Steps

> **CI checks only** (build + validate + test without regeneration): If the user asks to "run CI checks", "validate" or "check if a package passes all checks", skip to step 6. CI checks = build (`azsdk_package_build_code`) + validate (`azsdk_package_run_check`) + test (`azsdk_package_run_tests`).
Comment thread plugins/azure-sdk-tools/skills/README.md
| Skill | Triggers | Description |
| ----- | -------- | ----------- |
| [generate-sdk-locally](generate-sdk-locally/SKILL.md) | "generate SDK locally", "build SDK", "run SDK tests" | Generate, build, and test Azure SDKs locally from TypeSpec |
| [generate-sdk-locally](generate-sdk-locally/SKILL.md) | "generate SDK locally", "build SDK", "run SDK tests", "run CI checks", "passes all checks" | Generate, build, and test Azure SDKs locally from TypeSpec |
Comment thread .github/skills/README.md
| Skill | Triggers | Description |
| ----- | -------- | ----------- |
| [azsdk-common-generate-sdk-locally](azsdk-common-generate-sdk-locally/SKILL.md) | "generate SDK locally", "build SDK", "run SDK tests" | Generate, build, and test Azure SDKs locally from TypeSpec |
| [azsdk-common-generate-sdk-locally](azsdk-common-generate-sdk-locally/SKILL.md) | "generate SDK locally", "build SDK", "run SDK tests", "run CI checks", "passes all checks" | Generate, build, and test Azure SDKs locally from TypeSpec |
Comment thread .github/skills/azsdk-common-generate-sdk-locally/SKILL.md

## Steps

> **CI checks only** (build + validate + test without regeneration): If the user asks to "run CI checks", "validate" or "check if a package passes all checks", skip to step 6. CI checks = build (`azsdk_package_build_code`) + validate (`azsdk_package_run_check`) + test (`azsdk_package_run_tests`).
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

📊 GEPA Skill Quality Scores

Skill Quality Triggers Tests
azsdk-common-generate-sdk-locally ❌ 0.30 N/A ---
azsdk-common-pipeline-troubleshooting ❌ 0.34 N/A ---
azsdk-common-prepare-release-plan ❌ 0.37 N/A ---
azsdk-common-sdk-release ❌ 0.37 N/A ---
markdown-token-optimizer ❌ 0.39 N/A ---
azure-typespec-author ❌ 0.40 N/A ---
azsdk-common-apiview-feedback-resolution ❌ 0.43 N/A ---
skill-authoring ❌ 0.48 N/A ---
sensei ⚠️ 0.57 N/A ---

0/9 skills at quality ≥ 0.80

How to improve 
# Score a specific skill
python .github/skills/sensei/scripts/gepa/auto_evaluator.py score --skill <name> --skills-dir .github/skills --tests-dir tests

# Optimize a skill with GEPA
python .github/skills/sensei/scripts/gepa/auto_evaluator.py optimize --skill <name> --skills-dir .github/skills --tests-dir tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@praveenkuttappan praveenkuttappan praveenkuttappan approved these changes

@l0lawrence l0lawrence Awaiting requested review from l0lawrence

At least 1 approving review is required to merge this pull request.

Assignees

@l0lawrence l0lawrence

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug Bash] potential gap - if you ask the LLM to "run CI checks" it doesn't use the MCP at all

4 participants

@praveenkuttappan @l0lawrence